Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CT-271] [Feature] not_null test selects column instead of * #4777

Merged
merged 6 commits into from
Mar 10, 2022
Merged

[CT-271] [Feature] not_null test selects column instead of * #4777

merged 6 commits into from
Mar 10, 2022

Conversation

willbowditch
Copy link
Contributor

resolves #4769

Description

Currently the not_null test writes SQL equivalent to

SELECT *
FROM model
WHERE column IS NULL

Which on columnar databases like BigQuery makes running the test more expensive than

SELECT column
FROM model
WHERE column IS NULL

This PR changes the test to SELECT column, unless the store failures feature is enabled, in which case it is useful to keep the failing rows in totality using SELECT *.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change

@willbowditch
Only select target column for not_null test
7e0fbd3
@willbowditch
If storing failures include all columns in the select, if not, only s…
bd3a994 
…elect the column being tested

It's desirable for this test to include the full row output when using --store-failures. If the query result stored in the database contained just the null values of the null column, it can't do much to contextualize why those rows are null.
@cla-bot
Copy link

cla-bot bot commented Feb 24, 2022

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, don't hesitate to ping @drewbanin.

CLA has not been signed by users: @willbowditch

@cla-bot
Copy link

cla-bot bot commented Feb 24, 2022

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, don't hesitate to ping @drewbanin.

CLA has not been signed by users: @willbowditch

@willbowditch willbowditch marked this pull request as ready for review February 24, 2022 12:51
@willbowditch willbowditch requested a review from a team February 24, 2022 12:51
@willbowditch willbowditch requested review from a team as code owners February 24, 2022 12:51
@willbowditch
Copy link
Contributor Author

@drewbanin I have signed the CLA.

No changes to tests in the PR at the moment, but happy to have a go at adding some if they're needed.

@leahwicz
Copy link
Contributor

@cla-bot check

@cla-bot cla-bot bot added the cla:yes label Feb 24, 2022
@cla-bot
Copy link

cla-bot bot commented Feb 24, 2022

The cla-bot has been summoned, and re-checked this pull request!

ChenyuLInx
ChenyuLInx approved these changes Feb 25, 2022
View reviewed changes
Copy link
Contributor

@ChenyuLInx ChenyuLInx left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thanks for contributing this!!

@leahwicz leahwicz closed this Mar 9, 2022
@leahwicz leahwicz reopened this Mar 9, 2022
@leahwicz
Copy link
Contributor

leahwicz commented Mar 9, 2022

@willbowditch I had to close and reopen the PR to pick up on some new GitHub Checks that we just added. One is around how we do the Changelog and that's what failing here. So sorry for the inconvenience! Would you mind updating the Changelog entry to follow our new process that you can see documented here

@willbowditch
Copy link
Contributor Author

Hey @leahwicz - no worries, I've reverted the changes to the markdown changelog and pushed up the file changie created

@leahwicz leahwicz merged commit c251dae into dbt-labs:main Mar 10, 2022
@jtcohen6 jtcohen6 mentioned this pull request Jul 6, 2022
Closed
1 task
@joellabes joellabes mentioned this pull request Sep 22, 2022
Closed
4 tasks
@dbeatty10 dbeatty10 mentioned this pull request Nov 28, 2023
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ChenyuLInx ChenyuLInx ChenyuLInx approved these changes

Assignees
No one assigned
Labels
cla:yes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[CT-271] [Feature] not_null test selects column instead of *
4 participants
@willbowditch @leahwicz @ChenyuLInx @jtcohen6